Why change this? The change is same as the current implementation.

because the value of "firstIndex" is logged below

Revert this unnecessary change.

fixed

In a scenario where eviction does not occur, should overrideType also be storageInfoType?

I did it this way in order to not change the behavior of the code on the evict=false

this is code is really tricky, because it handles two things at once:

• the wanted "storageInfoType" from the policy
• the available types for the location (see the 'switch' on location.getStorageInfo....))

in theory the value for location.getStorageInfo.getType should be the same as storageInfoType when evict=false, but not for MEMORY

code is here:

val tryCreateFileTypeIndex =
      if (evict) {
        0
      } else {
        // keep the old behavior, always try to use memory if worker
        // has configured to use memory storage, because slots allocator
        // will not allocate slots on memory storage
        if (order.exists(_.contains(StorageInfo.Type.MEMORY.name()))) {
          order.get.indexOf(StorageInfo.Type.MEMORY.name())
        } else {
          order.get.indexOf(
            partitionDataWriterContext.getPartitionLocation.getStorageInfo.getType.name())
        }
      }

I am not still super familiar with this code (even if I debugged it many many times 🙄 ) and I did not want to break it

Suppose location.getStorageInfo.getType is set to HDD and createFileOrder is [HDD, S3], if the local HDD fails to create a file, storageInfoType will switch to S3. See following code segment:

for (i <- tryCreateFileTypeIndex until maxSize) {
  val storageStr = order.get(i)
  val storageInfoType = StorageInfo.fromStrToType(storageStr)
  val file = tryCreateFileByType(storageInfoType)
  if (file != null) {
    return file
  }
}

However, when storageInfoType is set to S3, the method createDiskFile still uses location.getStorageInfo.getType as HDD, which seems incorrect. Therefore, StorageType seems should always be overrideType.